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METHOD AND APPARATUS FOR MODIFYING A PORTION OF AN 
IMAGE FRAME IN ACCORDANCE WITH COLORIMETRIC 

PARAMETERS 



5 CROSS-REFERENCE TO RELATED APPLICATION(S) 

Reference is made to commonly assigned copending application 
Serial No 10/401,923, entitled "Method and System for Modifying Digital Cinema 
Frame Content" and filed on 28 March 2003 in the names of David L. Patton, Dale 
Mclntyre, John R. Fredlund, Michael E. McCrackan, Carlo V. Hume, and Arthur 
10 J. Cosgrove. 



FIELD OF THE INVENTION 

The present invention generally relates to colorimetric adjustment 
of defined areas within an image frame and more particularly relates to a color 
15 imaging system for selectively adjusting flesh tones within an image frame of a 
digitized motion picture in accordance with viewer preference. 



BACKGROUND OF THE INVENTION 

The difficulty of capturing and reproducing accurate and pleasing 
20 flesh tones is widely recognized in the imaging arts. For example, color 

negative/positive photographic systems in use today are designed to produce 
pleasing prints for most of the people in a target population. The print appearance 
includes both pleasing tone and color reproduction to produce colorful prints with 
good contrast, and particularly excellent flesh tone reproduction. Typically, 
25 existing photo systems are designed to be optimized for a particular skin type and 
preference. For example film colorimetric parameters can be optimized for 
capture and reproduction of some segment of the full range of flesh tones of world 
populations, such as Caucasian, Oriental, Asian, Indian, or Black flesh tones. 
Necessarily, photographic film, paper, and printer sets-ups are designed for 
30 providing pleasing color for a particular market segment, having a subset of the 
full range of flesh tones. As a result, a system optimized for producing 



photographs of people who have a lighter flesh tone may yield disappointing 
results for producing photographs of people who have a darker flesh tone. This 
can be especially true for the same film type and techniques used to photograph 
people whose flesh tones lie on the extremes of the range. This can result in loss 
5 of overall image quality, particularly with respect to facial features. 

Digital imaging now allows the capability to make colorimetric 
adjustments that can compensate for local differences in flesh tone reproduction. 
With digital images, such as those obtained from a digital camera or from a 
scanned print, colorimetric response of an imaging or display system can be 

1 0 adapted to allow an operator or consumer to modify flesh tone reproduction in 
order to provide a more pleasing output. For example, commonly assigned U.S. 
Patent No. 6,396,599 (Patton et al.) entitled "Method and Apparatus for Modifying 
a Portion of an Image in Accordance with Colorimetric Parameters" discloses a 
system that locates a flesh tone area in an image and allows an operator to adjust 

1 5 colorimetric attributes of that area in order to provide a more satisfactory output 
image. Colorimetric analysis and various supporting tools such as face 
recognition algorithms can be employed with such a technique in order to provide 
efficient identification of flesh tones in a scanned print or digital camera image. 

While methods such as those of the Patton et al. '599 disclosure 

20 present capable solutions for handling individual photographic images, the 

problem of flesh tone adaptation in motion picture imaging is more difficult. In 
conventional motion picture film processing, various editing procedures may be 
executed on a digitized image, scanned from the original film shot at the studio. 
After editing, the final image can then be rewritten onto a print film for 

25 distribution to local theaters and exhibitors. Thus, for conventional motion picture 
film production, editing personnel can perform colorimetric operations on a frame- 
by- frame basis, using the scanned digital image data. Using a sequence of frame 
image digitization, object and outline detection and masking, selective 
colorimetric modification, and printing to film, it is possible for editing personnel 

30 to adjust or to add color in successive frames of a conventional film motion 

picture. This overall sequence is used, for example, in colorization of black-and- 



f 



white motion picture film, using techniques such as those disclosed in U.S. Patents 
Nos. 5,912,994 (Norton et ah); 5,050,984 (Geshwind); and 3,784,736 (Novak). 

The introduction of digital cinema is expected to bring about 
numerous changes in how motion pictures are produced and distributed. Referring 
5 to Fig. 1 , there is shown a block diagram of a digital cinema preparation and 

distribution system 100 for providing motion picture images from a studio 1 10 to 
a post production facility 111, which digitizes the motion picture images and 
provides the digitized images over a transmission system 130 to an exhibition 
system 140, typically a movie theater. (The post production facility 111 may be 

10 under the control of the studio 1 10, or it may exist (as it typically would) as a 

separate entity in the overall system 100.) At post production facility 111, digital 
mastering is performed on film 1 12 that contains image content, such as the film 
feature, advertising, trailers, and the like. A datacine system 114 transforms the 
film content into digital image content and provides the digital image content to a 

1 5 rendering system 1 1 8, typically supported by a disk array 120, that renders the 
motion picture image data in a resolution suitable for distribution and display. 
Rendering system 118 may also accept input from auxiliary input devices 116 
such as data tape, D5 video tape, and DataCam devices. The fully mastered digital 
cinema output then is provided as a data stream to transmission system 130, which 

20 may utilize a satellite 138 in communication with a transmitter 122 connected to 
post production facility 111 equipment. Other alternative transmission media 
include a fiber cable connection 136, or transmission using an optical medium 
134, such as DVD or optical disks, or using a magnetic medium 132, such as data 
tape. At exhibition system 140, the mastered digital image data is received, such 

25 as at a receiver 148, an optical media reader 144, a magnetic media reader 142 or 
over a data or fiber optic cable connection 136. A cinema operating system 146, 
typically supported by disk array 120, accepts the digital input data, processes the 
input data stream for presentation, and provides this data for image forming and 
projection by one or more digital projectors 150. 

30 As the data path of Fig. 1 shows, digital cinema affords expanded 

opportunities for colorimetric manipulation of both a motion picture feature and 



its accompanying promotional content, such as advertising and trailers shown 
before or after the film. Certainly, a cinematographer, particular about achieving 
an artistic effect with a motion picture feature, may be sensitive to allowing 
adjustment of flesh tone or other colorimetric parameters by others at various 
5 points in the digital image data distribution stream or may want to put strict limits 
on such adjustments. An advertiser, on the other hand, may find it particularly 
desirable to adjust colorimetric attributes of advertising or trailer content to obtain 
broader and more effective distribution. For example, it may be of perceived 
benefit to a soft drink advertiser to distribute identical promotional image content 

10 to numerous sites, while allowing the operator at each site to adjust flesh tone or 
other colorimetric characteristics that appear within motion picture frames. Thus, 
the same commercial image content, playing at opposite corners of the world, can 
be given entirely different skin coloring at each projection site. 

Thus, what is needed is an apparatus and method for selectively 

1 5 adjusting the colorimetric attributes of flesh tone or changing other local color 
characteristics within an image frame of a digitized motion picture. 



SUMMARY OF THE INVENTION 
It is an object of the present invention to allow the modification of 
20 flesh tones, or of other selectable color areas, of a digital motion picture. With 

this object in mind, the present invention provides a method for modifying at least 
one colorimetric attribute of a predetermined region of a motion picture frame 
comprising: 

(a) providing metadata associated with the motion picture frame, said 
25 metadata defining the predetermined region of the frame; and 

(b) applying a colorimetric transform to pixels within the predetermined 
region, modifying the at least one colorimetric attribute thereby. 



30 



It is a feature of the present invention that it employs colorimetric 
transforms to modify selected portions of successive motion picture image frames 
in order to adapt the color of objects in the motion picture to specified preferences. 
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It is an advantage of the present invention that it allows color 
modification of image data at more than one point in the motion picture 
production chain. 

It is a further advantage of the present invention that it allows a 
5 local exhibitor to adapt colorimetric characteristics of flesh tones to suit local 

preferences. This capability can also be extended to other identified objects in the 
motion picture frame. 

These and other objects, features, and advantages of the present 
invention will become apparent to those skilled in the art upon a reading of the 
10 following detailed description when taken in conjunction with the drawings 

wherein there is shown and described an illustrative embodiment of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a schematic block diagram of a digital cinema preparation 
15 and distribution system. 

FIG. 2 is a plane view showing an example image frame having 
areas requiring colorimetric modification. 

FIG. 3 is a plane view showing how masks and/or coordinates can 
be used to specify image frame areas that are candidates for colorimetric 
20 modification. 

FIG. 4 is a schematic block diagram showing a colorimetric 
modification display apparatus of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
25 The present description is directed in particular to elements 

forming part of, or cooperating more directly with, apparatus in accordance with 
the invention. It is to be understood that elements not specifically shown or 
described may take various forms well known to those skilled in the art. 

Referring to Fig. 2, there is shown a sample image frame 20 within 
30 a sequence of image frames for a digital motion picture. The digital motion 

picture may be a feature film or may have promotional content, such as a trailer or 



advertising, for example. Within image frame 20 are two flesh tone regions 22, 
24. As was noted in the background, it can be desirable to modify flesh tone 
regions 22, 24 in order to be more appealing to a local audience. 

With digital cinema, it is possible for studio 110 (Fig. 1) to provide 
some amount of metadata corresponding to each image frame 20. Such metadata 
may refer without limitation to such areas or regions as flesh tone areas, hair color 
areas, eye color areas, and such other scene areas as grass and sky, and the 
presently-described algorithm may be used to modify a colorimetric attribute of 
such areas. For these areas, such as a flesh tone area, this metadata would include 
information identifying the area in some way. Referring to Fig. 3, there are shown 
two masks 32 and 34 that correspond to flesh tone regions 22 and 24 in Fig. 2. 
Metadata identifying masks 32 and 34 could be provided as bitmap data, such as 
in a binary form (for example, where "1" data indicates a pixel within masks 32 or 
34 and "0" data indicates a pixel outside these regions). Alternately, metadata 
could simply give coordinates 30 that define the outline of flesh tone regions 22 
and 24; Fig. 3 shows a few coordinate 30 points by way of example. Metadata 
could be provided in tabular form, as is shown in the simplified example of Table 
1. 

With respect to the digital image data stream, metadata could be 
supplied using the same communications channel used for the digital cinema 
content. Metadata might be interleaved with image data or provided before or 
after image content. Alternately, image data and metadata could be provided 
using separate data channels. 
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Table 1. Metadata Example for Successive Image Frames 20 



Frame # 


Object 


Bounding coordinates 


8043 


Flesh tone area, face, male 


(137,144) 






(433,158) 






(501, 199) 






(658,247) 






(661,398) 






(140,387) 


8044 


Flesh tone area, face, male 


(138,112) 






(440,208) 






(511,239) 






(677,267) 






(691,404) 






(138,415) 


8045 


Flesh tone area, face, male 


(138,111) 






(444,208) 






(511,259) 






(688,267) 






(706,411) 






(138,415) 
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Referring to Fig. 4, there is shown an image modification and 
display apparatus 60 for altering flesh tones or other color areas of digital cinema 
content at the exhibition site according to the present invention. As was shown in 
Fig. 1 , image data is provided to a server, which is identified in Fig. 4 as server 40, 
5 that is at, or communicates with, digital projector 150 at the local exhibitor site. 
Also provided to server 40 is the metadata identifying flesh tone areas or other 
areas to be modified. A control logic processor 48, such as a dedicated computer 
workstation 42 with an optional display monitor 44, maintains an optional library 
50, which can be a database of color transform Look-Up Tables, LUTs 46. For 

10 modifying the identified flesh tone areas in a frame 20, control logic processor 48 
selects an appropriate LUT 46 supplied to server 40 for use in the image data path. 
Alternately, LUT 46 may be provided directly to digital projector 150 for 
processing pixels within masks 32, 34 (Fig. 3) or other identified regions. LUT 46 
may be applied to the identified image data "on the fly" as it is being received or 

15 as it is being projected, or may be applied to stored image data maintained by 
server 40. The task of applying LUT 46 or other suitable transform may be 
performed within server 40 or within projector 150, depending on the processing 
capability available at either device. It must be noted that the configuration of Fig. 
4 shows only one possible embodiment; there are numerous possible arrangements 

20 that could combine the functions of server 40 and projector 1 50 for providing the 
necessary image processing, including application of LUTs 46 and related 
transforms. 

Control logic processor 48 may provide an operator with the option 
to select attributes of the color transform used as LUT 46. Alternately, LUT 46 

25 itself could be provided or specified along with the metadata for use by server 40. 
For example, an advertising distributor may decide to automatically download as 
metadata or specify a LUT 46 suitable for flesh tone modification when the 
advertising content is shown in any region of the world. In this way, a different 
LUT 46 can be applied to the same image data, depending on where content is 

30 shown, allowing suitable flesh tone transforms to be applied for the same content 
whether it is displayed in India, Brazil, or Finland, for example. 
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The listing of Table 1 is intended to be illustrative only; alternate 
and/or additional metadata fields could be provided for identifying an area of 
frame 20 that can be modified using alternate colorimetric transforms. The data 
format of the metadata could be a simple, open data format, such as employing 
5 familiar comma-separated fields, for example. More complex encoding schemes 
could be used, such as those employing compression, security algorithms wherein 
a key is required for decoding, or other known mechanisms for data transfer. 

Methods for applying LUT 46 as a colorimetric transform are well 
known in the art. It must be emphasized that other types of colorimetric 

10 transforms and algorithms based on, e.g., multi-dimensional (e.g., 3D) look up 
tables, matrices, and so on, are available for modifying color characteristics of an 
area of an image, well known to those skilled in the color imaging arts. 
Furthermore, a transform or algorithm may be selected for application to the 
whole image, if the transform or algorithm would primarily change skintones 

1 5 within the frame, and only slightly affect other tones and colors. In effect, in that 
case, the colorimetric transform is applied to pixels within the whole frame. 

In an alternate embodiment, studio 110 does not provide metadata 
defining flesh tone regions 22, 24 or other areas of frame 20 that may be subject to 
colorimetric transform. Instead, local exhibitor at exhibition system 140 may 

20 apply various types of imaging algorithms for detecting flesh tone regions 22, 24 
in the received image data. Utilities such as skin tone and facial feature 
recognition algorithms, well known in the imaging arts, could be used to identify 
flesh tone regions 22, 24 in order to allow LUTs 46 or other suitable transforms to 
be applied. 

25 It must be re-emphasized that the specification of the present 

disclosure describes modification of flesh tone regions 22 and 24, as were shown 
in the examples of Figs. 2 and 3. However, it can be appreciated that the method 
of the present invention need not be confined to flesh tones, but can be more 
broadly applied to other types of colored regions or objects in a motion picture 

30 image frame. For example, there may be reasons to emphasize or to de-emphasize 
certain colors or objects for special effects with some audiences. Hair color or 
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other features could also be changed using the basic methods of the present 
invention. 

Metadata provided for flesh tone regions 22, 24 or for other areas 
of image frame 20 may alternately specify limits or boundary values for LUT 
5 application to these regions. In this way, for example, the range of flesh tone 

selections for one or more actors could be restricted in order to prevent unrealistic 
treatment. This would allow some measure of control of the skin tone range by 
studio 110. 

Using the method of the present invention, studio 110 could 

10 provide exhibition system 140 with image data that is intended to be used as a type 
of master motion picture. LUTs or other suitable transforms could then be 
provided along with the metadata from studio 1 10 or may be provided to 
exhibition system 140 by some other transmittal means. It is then the function of 
the exhibitor to ensure that the master motion picture is suitably treated at the 

1 5 exhibition site before proj ection. 

Also using the method of the present invention, a single frame can 
be modified or the modification can be applied to a set of consecutively displayed 
motion picture frames, as would often be the case in a motion picture 
modification. In the latter case, it is important that the colorimetric adjustments 

20 from one frame to the next are done in a consistent manner for objects persisting 
from one frame to the next. As an example, for the best results for skintone 
reproduction of different races, different transforms might be used for humans in 
each frame having different skintones (e.g., asian, Caucasian, or african-american 
skintones, and so on). It would be important for each human in successive frames 

25 to be adjusted in the same way or a person's reproduction could shift from frame 
to frame. This is an additional desirable constraint, since it could be difficult if a 
facial recognition algorithm, or a skintone recognition algorithm, were to be used 
unless these algorithms were capable of identifying specific individual skintone 
types. 

30 The transforms used to modify the imagery in selected areas need 

not be limited to color transformations. Other modifications to the imagery can be 
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supported, e.g., transformations that would affect sharpness, tonescale, color 
balance, and the like. For example, if an undesirable object, such as an 
advertisement, is present in the scene and there is a desire to obscure the object, a 
transform may be applied which blurs the selected pixels representing the object. 
5 This technique may also be used to obscure, i.e., unsharpen or throw out of focus, 
distracting objects and faces. For example, this may be especially useful for 
providing a more enjoyable viewing experience of an image of an emphasized 
person or couple, such as a bridal couple, if other faces in the image are 
deemphasized. In general, such a method modifies at least one attribute of an 

10 object by the steps of (a) providing metadata defining a predetermined region 
containing the object; and (b) applying a transform to pixels within the 
predetermined region, thereby modifying the attribute of the object whereby the 
object is treated differently relative to other objects in the frame. 

The invention has been described in detail with particular reference 

15 to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the scope of the invention as described 
above, and as noted in the appended claims, by a person of ordinary skill in the art 
without departing from the scope of the invention. For example, various types of 
imaging transforms could be applied at different stages of production for digital 

20 motion picture content, in addition to the LUTs 46 described. The method of the 
present invention could be applied in preparation of conventional film-based 
motion pictures, but would be applied at an editing facility, allowing distribution 
of the same image content having different colorimetric transforms applied to suit 
local/regional preferences. 

25 Thus, what is provided is an apparatus and method for selectively 

adjusting flesh tones or other areas within an image frame, or to an entire image 
frame, of a digitized motion picture in accordance with viewer preference. 
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PARTS LIST 


20 


Image frame 


22, 24 Flesh tone regions 


30 


Coordinates 


32, 34 Masks 


40 


Server 


42 


Workstation 


44 


Display monitor 


46 


Look-Up Table (LUT) 


48 


Control logic processor 


50 


Library 


60 


Image modification and display apparatus 


100 


Digital cinema preparation and distribution system 


110 


Studio 


111 


Post production facility 


112 


Film 


114 


Datacine system 


116 


Auxiliary input devices 


118 


Rendering system 


120 


Disk array 


122 


Transmitter 


130 


Transmission svstem 


132 


Magnetic medium 


134 


Optical medium 


136 


Fiber cable connection 


138 


Satellite 


140 


Exhibition system 


142 


Magnetic media reader 


144 


Optical media reader 


146 


Cinema operating system 


148 


Receiver 



-13- 

1 50 Digital projector 



